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ABSTRACT 


This  thesis  exaiines  spectronetric  oil  analysis 
data  in  an  attempt  to  construct  tables  of  statistical 
estimates  for  use  in  evaluating  a  laboratory's 
performance  individually  and  in  comparison  to  a 
control  laboratory.  Tables  of  estimates  were  obtained 
from  data  provided  ny  twenty  six  laboratories. 
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I.  INTRODUCTION 


la  1956,  the  Naval  Air  Rework  Facility  at  Pensacola 
started  a  trial  program  to  determine  if  spectrometric 
analysis  of  oil  samples  could  be  used  to  predict  aircraft 
engine  failures.  The  Nival  Oil  Analysis  Program  (NOAP) 
evolved  as  a  conseguence  of  the  success  of  the  trial 
program.  The  program  has  been  expanded  to  include 

monitoring  virtually  all  Navy  lubricated  systems.  In  1976, 
NOAP  was  merged  with  similar  Army  and  Air  Force  programs  and 
became  the  Joint  oil  Analysis  Program  (JOAP)  .  References  1 
and  2  provide  a  more  detailed  background  of  the  oil  analysis 
program. 

The  prediction  of  a  pending  equipment  failure  is 

facilitated  by  the  spectrometric  oil  analysis  of  a  sample  of 
the  lubricating  fluid  from  the  equipment.  The  fluid  sample 
is  burned  in  the  spectrometer  and  the  concentrations  of 
certain  individual  wearmetal  elements  in  parts  per  million 
(PPH)  are  determined  by  the  wavelengths  of  the  light 
emitted.  A  record  maintained  for  each  equipment  contains 
the  results  for  each  wearmetal  concentration  from  previous 
samples.  After  a  sample  from  a  particular  piece  of 

equipment  is  burned,  an  evaluator  reviews  the  results  of 
current  and  past  burns  to  determine  if  there  is  an  abnormal 
trend  development  or  abnormal  concentration  level. 

Depending  on  which  wearmetals  or  combination  of  wearmetals 
have  developed  an  abnormal  trend,  the  evaluator  is 
frequently  able  to  pinpoint  the  source  and  recommend  the 
required  preventive  maintenance  action  to  the  equipment 
custodian.  Reference  3  contains  procedures,  intervals  for 
sampling  and  normal  limits  of  wearmetal  concentrations  for 


some  of  the  equipments  monitored  by  JO  41?. 


The  normal  liaits  of  wearmetai  concentrations  are  not 
precise  boundaries.  oae  abnormal  sample  from  a  given 
equipment  does  not  necessarily  constitute  a  requirement  for 
a  aaiatanaace  action.  Hany  factors  must  be  considered  by 
the  evaluator.  Typically  a  shorter  sampling  interval  or  an 
immediate  resample  aay  coafirm  an  abnormal  level  or  aay  lead 
the  evaluator  to  conclude  that  soae  error  has  occurred 
(sample  comtaminat ion  or  sample  interchange)  .  Other  factors 
are  the  tolerances  within  which  a  laboratory  can  obtain  the 
same  results  on  repeated  experiments  (repeatability) ,  and 
can  reproduce  either  its  own  or  another  laboratory's  results 
(reproducibility)  .  These  two  factors  are  considered  in  this 
thesis. 

Intuitively,  it  can  oe  seen  that  a  laboratory  must  be 
capable  of  obtaining  fairly  consistent  results  for  repeated 
burns  of  an  oil  sample.  Otherwise  there  would  be  little 
reason  to  expect  the  laboratory  to  provide  data  from  which 
the  evaluator  would  detect  a  discrepant  equipment.  We  might 
expect  also-  that  the  procedure  would  often  identify  a 
properly  functioning  jquipment  as  being  discrepant. 
Similarly,  because  military  equipments  are  frequently 
transferred  from  one  location  to  another,  it  is  desirable  to 
be  able  to  use  the  results  from  different  laboratories  for 
the  same  equipment  without  having  to  wait  for  each 
laboratory  to  separately  develop  trend  information  about 
that  equipment.  4s  a  means  of  insuring  that  consistent 
results  are  provided  by  laboratories,  a  certification 
program  exists  in  JOAP  (reference  3)  .  The  procedures  for 
certification  consider  only  an  evaluation  of  a  laboratory's 
individual  performance.  Reference  h  contains  recommended 
procedures  for  laboratory  certification  involving  both  an 
evaluation  of  a  laboratory's  performance  and  its  comparison 
with  a  control  laboratory.  Table  8-2  of  reference 
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contains  values  of  maximum  allowable  accuracy  index  and 
repeatanilit y  index  for  laboratory  certification.  But 
questions  nave  arisen  as  to  whether  the  values  in  the  table 
refer  to  repeatability  or  reproducibility  and  what  is  the 
significance  level  of  the  values.  Because  of  these 
questions  there  is  some  doubt  about  the  validity  of  the 
numbers. 

This  thesis  investigates  data  provided  by  twenty  six 
laboratories  that  utilize  the  Baird  Atomic  AE35/a-3  Atoaic 
Emission  Spectrometer.  The  data  collected  are  for  0,  3,  10, 
50  and  100  PPM  concentrations  for  each  of  twenty  elements. 
Each  sample  analysis  is  replicated  fifteen  times  and  each 
set  of  fifteen  replications  is  repeated  on  two  different 
days. 


II.  CONSIDERATIONS  IN  SPEC ISOMETRIC  OIL  ANALYSIS 


A.  MEASUREMENT  ERRORS 


As  discussed  in  reference  5,  there  are  numerous 
potential  sources  of  error  in  results  from  the  atomic 
emissions  spectrometer.  Per  each  observation  taicen  (for 
each  oil  sample  burned)  r  the  obsarved  reading  can  be 
considered  to  include  a  true  PPM  reading  plus  an  error  term. 
The  error  term  can  be  modeled  as  a  random  variable  and,  for 
the  moment,  can  be  thought  of  as  an  accumulation  of  the 
effects  of  all  the  possible  sources  of  er'  or.  These  sources 
include  inputs  such  as  temperature,  humidity,  electrode  gap 
width,  spectrometer  standardization,  inhomogeneous  oil 
samples,  operator  technique,  contamination,  etc.  The 
observed  reading  can  be  mathematically  represented  by: 


Y.  »  U  +  E.  (1) 

1  I 

where  Y  is  the  result  of  the  ith  observation, 
i 


U  is  the  true  PPM,  and 

E  is  the  error  component  of  the  ith  observation, 
i 

Many  of  the  error  inputs  can  be  controlled  io  some 
degree  by  careful  attention  to  precedures  established  for 
the  Oil  Analysis  Program.  These  include  sample  handling, 
spectrometer  standardization  and  operator  techniques. 


However,  in  burning  oil  samples  collected  from  operating 
equipments,  the  operator  has  no  prior  estimate  of  the  true 
PPM  of  the  sample  nor  of  the  error. 

From  aquation  (1),  it  can  be  seen  that  if  tha  magnitude 
of  the  random  error  is  allowed  to  be  large  relative  to  the 
true  PPM,  the  error  could  mask  a  significant  change  in  the 
estimated  PPM.  Since  the  error  is  not  measurable,  some 
method  is  needed  to  measure  the  performance  of  a 
spectrome tar. 


B.  MEASURES  OF  PSRF0RMAH3E 


Since  the  random  error  term  can  be  considered  to  be 
normrlly  distributed  (references  4,  5  and  S) ,  good  measures 
of  spectrometer  performance  could  be  expected  to  involve 
functions  of  the  sample  mean  and  sample  variance.  To  test 
whether  a  given  spectrometer  is  producing  results  within  an 
acceptable  error  tolerance  level,  one  might  make  several 
observations  on  an  oil  sample  of  known  PPM  concentration  and 
compute  the  sample  mean  and  sample  variance  of  the 
spectrometer  readings.  If  the  computed  quantities  were 
within  some  predetermined  bounds,  the  spectrometer  could  be 
considered  to  be  operating  satisfactorily.  Dthecwise,  the 
spectrometer  should  be  realigned  electronically 
(restandardized)  and  retested.  The  measures  of  performance 
currently  used  in  JOAP  ace: 

A I  =  i  u  -  Yl  (2) 

and 

2  l  /2 

ax  =  (SUM.(y.  -  y)  /  (N- 1 ) )  ,  i  =  (3) 

a  i 


11 


vi'MM 


where  n  is  the  known  PP5  concentration  of  the  oil  sample. 


y  =  SUB  (y  /N)  ,  i  =  1  , .  . .  ,  N ,  is  the  sample  mean, 
i  i 

y  is  the  ith  observation  on  the  sample, 
i 

N  is  the  total  number  of  observations  taken, 

AX  is  the  accuracy  index  (true  PPS  -  sampla  mean)  ,  and 

RI  is  the  repeatability  index  (sample  standard 

deviation)  . 

For  the  reasons  cited  ab^e,  it  was  decided  that  the 
measures  of  performance  represented  by  equations  (2)  and  (3) 
are  satisfactory  for  use  with  oil  analysis  program  data. 

Currently  used  values  of  the  maximum  allowable  AI  and  RI 
are  given  in  table  8-2  of  reference  3.  In  their  work  with 
data  obtained  form  the  oil  analsis  program,  D.  R .  Barr,  T. 
Jayachanaran ,  and  H.  J.  Larson  have  found  that  the  tabled 
values  may  not  be  realistic  bounds  (reference  4) . 

As  current  doctrina  in  the  JOAP  procedure,  an  operator 
makes  ten  sample  burns  from  a  standard  oil  sample  of  known 
PPH  concentration  and  uses  equations  (2)  and  (3)  to  compute 
AI  and  SI  for  each  element  under  analysis.  If  the  computed 
AI  and  RI  do  not  exceed  tabled  values,  the  laboratory  is 
considered  to  be  operating  within  acceptable  tolerances. 
Otherwise,  the  operator  should  restandardize  the 
spectrometer  and  repeat  the  sample  burns. 


12 


'  f  . 

.V?- 


* 


III.  ANALYTICAL  RESULTS 


A.  OBJECTIVES 


Tha  goal  of  this  project  was  to  statistically  estimate 
the  bounds  or  limits  for  M  and  SI  for  both  tha  within 
laboratory  effect  (repeatability)  for  a  single  laboratory, 
and  between  laboratory  effect  (reproducibility)  for  a  given 
laboratory  compared  with  a  control  laboratory.  However,  it 
was  discovered  that  three  separata  and  distinct  sets  of 
indicias  could  be  identified.  Appendix  B  is  devoted  to  a 
development  of  the  th.reo  sets  of  bounds  for  the  indices.. 
Notationally ,  the  three  situations  are  identified  as  Case  I, 
Case  II  and  Case  III. 


B.  RESULTS 


The  procedures  presented  in  this  section  apply  to  any 
one  of  the  twenty  elements  for  which  tha  AE35/U-3 
spectrometer  is  used  in  JOAP.  They  concern  standardization 
with  standard  oil  samples  having  any  one  of  0,  3,  10,  50  or 
"".00  PPM  concentrations.  The  procedures  should  be  applied 
for  each  element  and  concentration  of  interest. 

1 .  Case  I 


Case  I  corresponds  to  the  current  usage  of  &l  and 
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EX,  that  is,  a  Laboratory  Bakes  a  set  of  H  sample  burns  with 
a.  standard  oil  sample  and  computes  All  and  BIT  using 
equations  (2)  and  (3)  .  The  computed  quantities  (using  n  = 
10  or  n  »  15  sample  burns!  may  be  compared  with  the  bounds 
contained  in  table  I  or  IV  respectively  of  +-he  next  section. 
If  the  computed  quantities  are  not  greater  than  the  tabled 
values,  the  laboratory's  spectrometer  may  be  considered  to 
be  operating  within  acceptable  tolerances  for  Case  I. 

All  is  the  magnitude  of  the  deyia.tion  of  the  sample 
mean  from  the  known  PPM  concentration  and  might  be  viewed  as 
an  indicator  of  spectrometer  alignment  or  standardization, 
ail  is  a  measure  of  the  variability  in  the  spectrometer's 
observations.  Excess  variability  suggests  either  poor 
operator  technique  or  an  erratic  spectrometer  channel. 

2.  Case  II 

Case  II  may  be  considered  to  be  the  tolerances 
within  which  a  given  laboratory  would  be  expected  to 
reproduce  its  own  results  with  a  second  complete  set  of  H 
sample  burns  from  the  same  oil  sample.  The  laboratory  could 
complete  these  procedures  using  any  oil  sample.  The 
following  comput?. tions  would  be  made-  for  the  two  sets  of 
sample  burns: 

A 12  »  |y  -  xj  (4) 

and 

z 

BI2  =  (s  /s  )  (5) 

y  x 

where  y  is  the  sample  mean  computed  from  the  first  set  of 
sample  burns, 
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3c  is  the  sample  mean  computed  from  the  second  set  of 
sample  burns, 

s  is  Ell  computed  from  the  first  set  of  burns,  and 

y 


s  is  Ell  computed  from  the  second  set  of  burns, 
x 

The  computed  quantities  (using  N  =  10  or  N  =  15 

sample  burns)  from  aquations  (4)  and  (5)  may  be  compared 
with  the  bounds  given  in  Table  II  or  V  respectively  of  the 
next  section.  If  the  computed  value  of  AI2  from  equation 
(4)  is  not  greater  than  the  tabled  bound  and  if  the  computad 
value  of  R 12  fro®  equation  (5)  lias  between  the  reciprocal 
of  the  tabled  value  and  the  tabled  value,  the  laboratory  may 
be  considered  to  be  operating  within  acceptable  tolerances 
for  Case  II. 

AI2  Is  the  magnitude  of  the  deviation  of  sample 
means  between  sets  of  N  observations  and  might  be  viewed  as 
an  indicator  of  a  change  in  sp  ‘-.rtr  oaeter  standization .  RI2 
is  a  ratio  of  sample  variance  a  *c'.  oeasures  a  Laboratory‘s 
ability  to  reproduce  its  previous  results. 

A  word  of  caution  concerning  the  interpretation  of 
the  index  EI2  is  in  order.  During  the  analyses,  it  was 
noted  that  the  computed  sample  standard  deviations  from  data 
sets  taken  from  within  a  spectrometer  appeared  to  be 
related.  The  correlation  coefficent  (appendix  c)  was 
computed  for  each  PPM  concentration  and  element  combination 
and  tested  for  significance  (Table  VII) .  Because  the 
correlation  noted  was  in  general  quite  high,  it  should  be 
expected  that  in  almost  all  applications  of  Case  II 
procedures,  for  812,  the  laboratory  will  pass. 
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3.  Case  III 


Case  XXX  may  be  considered  to  be  the  tolerances 
within  which  a  given  laboratory  would  be  expected  to 
reproduce  the  results  obtained  by  a  control  laboratory. 
Under  Case  XXI  procedures,  both  the  control  laboratory  and 
the  laboratory  under  test  would  complete  N  sample  burns  on 
the  same  oil  sample.  The  following  computation  would  be 
made  for  the  two  sets  of  sample  burns. 

AI3  *  |y  -  x|  (6) 

and 

2 

RI3  =  (s  /s  )  (7) 

Y  * 

where  y  is  the  sample  mean  of  the  control  laboratory’s  N 
observations , 

x  is  tha  sample  mean  of  the  test  laboratory's  N 
observations , 

s  is  HI1  for  the  control  laboratory,  and 

y 


s  is  Ell  for  the  test  laboratory. 

x 

The  computed  quantities  (using  H  =  10  or  B  *  15  sample 

burns)  may  be  compared  with  the  bounds  given  in  table  XXX  or 
VI  respectively  of  the  next  section.  If  the  computed  value 
of  AI3  from  aquation  (6)  is  not  greater  than  the  tabled 
bound  and  if  the  computed  value  of  ai3  from  aquation  (7) 
lies  between  the  reciprocal  of  the  tabled  bound  and  the 
tabled  bound,  the  laboratory  under  test  may  be  considered  to 
be  operating  within  acceptable  tolerances  for  Case  III. 
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AI3  is  the  magnitude  of  the  deviation  between  a 
control  laboratory's  and  a  given  laboratory's  sample  leans 
and  night  be  viewed  as  an  indicator  of  a  given  laboratory's 
alignment  with  respect  to  the  control  laboratory.  BI3  is  a 
ratio  of  sample  variances  and  measures  a  laboratory's 
ability  to  reproduce  the  control  laboratory's  results. 

C.  TABLES 

Tables  I  through  VTI  are  discussed  in  section  III  and  in 
appendices  B  and  Z.  Table  VIII  is  a  comparison  of  the 
number  of  laboratories  for  which  the  data  submitted  was  used 
with  tl  dumber  of  laboratories  submitting  data  (see 
discussio,  .  Table  II  is  an  example  of  the  computer  output 
for  oue  of  the  100  analysis  of  variance  problems  (100  PPM 
and  element  comb inations)  .  Appendices  A,  B  and  C  have 
developments  for  aost  of  table  IX.  Tables  I  through  VII  are 
taken  from  t.he  100  computer  output  pages.  Table  X  is  also 
taken  from  the  computer  output  pages  and  shows  for  which  PPM 
and  element  combinations  the  spectrometer  effect  was 
significant  (appendix  A)  . 
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TABLE  VIII 


SJO*  OF  LABS  USED /NO.  OF  LABS  SUBMITTED 


P°M 

0  - 

3 

FE 

25/2  5 

2  6/26 

AG 

25/25 

26  /  26 

AL 

24/25 

23/23 

BE 

21  /2L 

22/22 

CR 

25/25 

26/26 

CU 

25/25 

2  6/26 

MG 

25/25 

26  /26 

NA 

22/22 

20/23 

NI 

25/25 

26/26 

pe 

24/24 

25/25 

St 

25/25 

26/26 

SN 

21/23 

19/24 

T I 

25/25 

26/26 

B 

21/23 

23/24 

BA 

22/23 

24/24 

CD 

22/22 

23/23 

MN 

21/22 

21/21 

NO 

25/25 

24/26 

V 

22/22 

20/23 

ZM 

19/21 

19/21 

10 

50 

100 

25/26 

26/26 

26/26 

26/26 

26/26 

26/26 

26/26 

26/26 

26/^6 

22/22 

21/22 

21/22 

26/26 

25/26 

26/26 

?,6  /26 

26/26 

26/26 

26/26 

26/26 

25/26 

21/23 

21/23 

19/23 

26  /26 

26/26 

26/26 

25/25 

24/25 

23/25 

26/26 

26/26 

26/2  6 

21/23 

23/25 

23/24 

26/26 

26/26 

25/26 

23  /24 

20/24 

22/24 

23/24 

23/24 

23/24 

22/23 

21  /23 

21/23 

22  /2  4 

21/23 

21/23 

25/26 

24/26 

24/26 

20/23 

19/22 

21/23 

19/21 

14/19 

14/20 
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TABLE  IX 


PPM:  0  ELEMENT:  FE 


Y  I J  .  BAR  STANDARD  DEVIATION 


LAB 

DAY  1 

DAY  2 

DAY  1 

DAY  2 

I 

-0.233 

0.  C60 

0.226 

0.2  03 

2 

0.053 

-0  .067 

0.368 

0.232 

3 

C.  093 

0.007 

0.194 

0  .428 

-0. 260 

-0.547 

0.210 

0.270 

3 

-0.313 

-0.620 

0.181 

0.22  7 

6 

-0.333 

-0.C47 

0.238 

0.136 

7 

-0.667 

0.200 

0.488 

0.414 

b 

0.233 

0.300 

0.206 

0.169 

9 

0.347 

0.227 

0.213 

0.258 

11 

-C .300 

-0.320 

0.256 

0.462 

12 

-C.000 

0.153 

0.314 

0.484 

13 

G.480 

0.773 

0.457 

0.433 

14 

-0.3  73 

-0  .407 

0.128 

0.139 

15 

0.120 

-0.120 

0.273 

0.347 

16 

-0.540 

-0.687 

0.  192 

0.239 

17 

-0.067 

-0.200 

0.184 

0.1o5 

la 

-0.500 

0.240 

0.  125 

0.083 

19 

0.0 

0  .0 

0.0 

0.0 

20 

-G.  073 

-0.053 

0.313 

0  .242 

21 

0.0 

O.G 

0.0 

G.  0 

22 

-0.087 

-0.140 

0  .300 

0.282 

23 

0.013 

0.013 

0.164 

0.270 

24 

-0.153 

0.147 

0.136 

0.458 

25 

-0.347 

-0.247 

0.449 

0.400 

26 

-0.427 

Q.C60 

0.592 

0.447 

C-RANG  MEAN  (Y 

...  BAR) 

CQRR  COSF 

• 

C92 

0.630 

AuCi  VA 

TABLE 

SOURCE 

OF  SUM 

OF  SQUARES  MEAN  SJUAR 

MEAN 

1. 

6.366 

BETWEEN  INSTRUM 

24. 

48.553 

2.023 

DAY  WITHIN  INST 

25. 

L6.740 

0.670 

wITHI  N 

IN  STRUM 

700. 

60.971 

0.087 

TOTAL 

750. 

132.630 

SOURCE 

F  STATISTIC  F  VALUE  COMP 

.VAR .ES 

BETWEEN  IN  STRUM 

3.0213 

1.9643 

0.045 

DAY  WITHIN  INST 

7  .6875 

1.5219 

0.039 

WITHIN 

instrum 

0.087 

TABLE 

VALUES  FOR  10  BURNS 

CASE  I 

CASE  II 

CASE  III 

A  I 

C.615 

0.631 

0.869 

R  I 

C  .520 

4.026 

4.026 

TABLE 

VALUES  FOR  15  BURNS 

CASE  I 

CASE  I! 

CASE  III 

A  I 

C  .  oQ6 

0.615 

0.  857 

R I 

G  •  4  B  4 

2 .9  79 

2.979 
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TABLE  X 


S  IGNIFI 

:ant 

SPECTROMETER 

EFFECT 

PPM 

3 

3 

10 

50 

ICQ 

FE 

YES 

NO 

YES 

YES 

NO 

AG 

YES 

YES 

YES 

YES 

NO 

AL 

YES 

NO 

YES 

YES 

YES 

BE 

YES 

YES 

YE  S 

NO 

NG 

CR 

YES 

YcS 

YES 

NO 

NO 

C'J 

YES 

YES 

YES 

NO 

NO 

MG 

YES 

YE  S 

YES 

YES 

NO 

NA 

NO 

YES 

NO 

YES 

YES 

NX 

Y  ES 

NO 

NO 

YES 

YES 

PB 

NO 

NO 

NO 

YES 

NO 

SI 

NO 

NO 

NO 

NO 

NO 

SN 

Y  ES 

MO 

YES 

YES 

YES 

T1 

YES 

NO 

NO 

YES 

NC 

B 

NO 

NO 

NO 

NO 

YES 

EA 

Y  ES 

YES 

YES 

NO 

YES 

CO 

YES 

NO 

YES 

NO 

NO 

MN 

YES 

YE  S 

VES 

NO 

NO 

MQ 

YES 

NO 

YES 

YES 

MO 

V 

NO 

NO 

YES 

YE  S 

NO 

ZN 

YES 

NC 

NO 

YES 

NO 
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IV.  DISCUSSION  AND  CONCLUSIONS 


The  data  submitted  was,  in  general,  assumed  to  be  a 
representative  sampling  from  the  population  of  AE35/U-3 

spectrometers  used  by  JOAP.  However,  from  x  vs  s  plots 

(reference  9}  made  from  the  sample  means  and  sample  standard 
deviations  of  the  data,  it  appears  that  some  laboratories 
submitted  data  which  was  not  consistent  with  the  data  from 
other  laboratories.  Assistance,  in  determining  which  sets 
of  data  appeared  to  be  erratic  or  from  spectrometer  channels 
with  set  up  (standardization)  problems,  was  obtained  from 
the  JOAP  Technical  Support  Center.  Table  VIII  shows  the 
number  of  laboratories  from  which  the  data  was  used  compared 
with  the  number  of  laboratories  submitting  data.  In 

general,  only  those  sets  of  data  for  which  the  laboratory 

appeared  to  stand  apart  from  the  rest  on  the  x  vs  s  plot  and 
were  identified  by  the  Technical  Support  Center  were  removed 
from  tae  analyses.  It  is  felt  that  a  few  of  the 

laboratories  may  have  been  more  than  conscientious  in 
setting  up  the  spectrometer  for  the  sample  burns  while  a  few 
others  may  have  been  somewhat  careless.  If  this  were  indeed 
the  situation,  the  variability  (caused  by  the  two  extremes) 
could  have  been  responsible  for  the  apparent  correlation 
noticed  (table  VII)  for  many  PPH  concentrations  and  element 
combinations.  This  would  contribute  erroneously  to  the  sums 
of  squares  due  to  error,  making  the  estimates  larger  in 
magnitude  than  they  should  be. 

The  tables  of  estimates  are  in  a  crude  form:  perhaps 
smoothing  over  elements  and  PPM  concentrations  could  reduce 
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the  coarseness  of  the  values.  To  use  the  tables  for  PPM 
concentrations  between  those  given,  the  user  might  use 
either  linear  interpolation  or  least  squares  regression. 
Linear  least  squares  regression  was  trie!  using  PPM  as  the 
independent  variable  ani  the  estimate  as  the  dependent 
variable.  In  general,  a  good  fit  was  obtained.  However, 
some  of  the  tabled  values  appear  to  be  outliers  and  in  those 
cases  the  results  were  not  usable.  Using  Bartlett's  test 
for  equal  variances  (appendix  B)  ,  it  was  found  that  the 
assumption  of  equal  (homogeneous)  variances  was  not  valid. 
According  to  Hetec  and  Wasserman  (reference  9) ,  unequal 
variances  can  have  pronounced  effects  on  inferences  about 
the  variance  components  (appendix  A)  when  using  a  random 
effects  model. 

wuture  work  in  this  area  should  include  a  repeat  of  this 
project  with  a  qualified  observer  present  at  each  laboratory 
for  the  data  gathering  to  ensure  that  proper  standardization 
procedures  are  followed.  In  this  project,  it  was  found  that 
the  day  (standardization)  effect  was  significant  for  all  PPM 
and  element  combinations.  In  many  instances,  the  P 
statistic  was  more  than  ten  times  larger  than  the  tabled  ? 
value  (appendix  A)  . 

Because  the  day  effect  was  significant  in  all  cases,  it 
is  felt  that  the  Case  II  procedures  described  in  this  report 
could  be  very  useful  to  JOhP.  case  I  procedures  (currently 
used  in  JGAP)  used  in  conjunction  with  case  II  and  case  III 
procedures  might  ensure  that  laboratories  could  produce 
consistent  results  on  both  interlaboratory  and 
intralah oratory  comparisons. 
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APPENDIX  A 


MATHEMATICAL  MODEL 


An  Analysis  of  Variance  Modal  (references  7  and  9)  is  a 

statistical  tool  utilize!  by  an  experimenter  to  study  the 

relationship  between  a  dependent  variable  (an  observation  on 

a  sample  burn)  and  ona  or  more  independent  variables  (mean 

PPM  concentration,  spectrometers,  standardization  setting, 

error,  etc)  .  No  assumptions  are  required  about  the  nature 

of  the  statistical  relation.  The  effects  of  the  independent 

variables  are  separately  studied  by  partitioning  the  total 

sum  of  squares  (the  summation  of  all  the  observations 

squared)  and  the  associated  degrees  of  freedom  into  smaller 

sums  of  squares  that  are  specifically  related  to  the 

independent  variables.  Dividing  a  sum  of  iuares  for  an 

effect  by  its  associated  degrees  of  freedom  gives  an 

unbiased  variance  estimate  (mean  square)  .  A  ratio  of  two 

variance  estimates,  under  the  null  hypothesis  that  the 

particular  effect  is  zero,  has  an  F  distribution  with 

degrees  of  freedom  v  (numerator  degrees  of  freedom)  and  v 

1  2 

(denominator  degrees  of  freedom)  .  At  a  chosen  level  of 

significance,  the  test  statistic,  F  (the  ratio  of  variance 

estimates),  can  be  tested  for  significant  effect  by 

comparing  F  with  the  tabled  value  of  an  F  distribution  with 

v  and  v  degrees  of  freedom,  A  value  of  F  greater  than  or 
1  2 

equal  to  the  tabled  value  (at  the  chosen  level  of 

significance)  implies  that  the  effect  is  significant; 
otherwise,  it  may  be  considered  to  be  zero.  Analysis  of 
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variance  models  nay  be  used  for  for  fixed  effects  (levels  of 
independent  variables  under  study  aia  the  only  ones  of 
interest)  or  for  random  effects  (levels  of  independent 
variables  under  study  are  a  subset  of  a  population  that  is 
of  interest)  . 

The  model  used  in  analyses  of  the  data  is  a  Nested 
Bandom  Effects  Model  (reference  7)  .  &  random  effects  model 
is  considered  appropriate  because  it  allows  '  for 
generalization  of  the  results  obtained  from  the  randomly 
selected  subset  of  a  population  (spectrometers)  of  interest 
to  the  entire  population*  Twenty  five  laboratories  were 
chosen  from  the  population  of  127.  The  laboratories  were 
selected  by  drawing  uniform  random  numbers  between  1  and  127 
until  twenty  five  laboratories  had  been  selected.  However, 
those  laboratories  based  aboard  ships  and  in  foreign 
countries  were  exempted  from  selection.  The  time  required 
to  mail  oil  samples  to  the  exempted  lano j.atories  and  for 
them  to  return  the  results  was  thought  to  be  excessive. 

In  the  random  selection  of  oil  laboratories  for 
participation  in  the  data  collection,  the  laboratory 
expected  to  be  the  control  laboratory  (JOhP  Technical 
Support  Center)  was  not  selected.  However,  it  was 
considered  desirable  to  have  the  control  laboratory  included 
in  the  sampling.  Hence,  data  were  also  collected  from 
Technical  support  Center,  making  twenty  six  laboratories  in 
total.  The  addition  of  the  non-raadomly  selected  laboratory 
to  the  sample  should  not  significantly  alter  the 
generalization  of  the  results  to  the  entire  population  of 
spectrometers.  The  nested  feature  of  the  model  allowed  the 
author  to  investigate  the  day  (standardization)  effect 
within  spectrometers. 

The  model  can  be  expressed  mathematically  as: 


# 


I  * 

ijk 

i  3  1 1  ■  •  •  ^  X 
where  Y 

ijk 

on  day  j. 


+  4  +  B 

i  j<i> 


♦  E 


ij)c 

»  j  ■*  1  »•  t  .  » J  ,  1C  3  1  ,  •  •  •  /  K 
is  the  k£h  observation  from  the  ith 


spectrometer 


u  is  the  mean  effect, 

&  is  the  i£h  laboratory  affect, 
i 


B  t  ^  is  the  jt&  day  affect  within  laboratory  i,  and 


E  is  the  random  error  of  the  kth  observation  on 
ijk 

instrument  i  and  day  j. 


The  assumptions  af  the  model  ara  that  E  is 

ijlc 

2  2 
distributed  NI(0,a  ),  A  is  distributed  NI(Q,o  ),  and  B 

1  A  j<i> 

2  2 
is  distributed  HI(0,ar  )  for  each  i.  The  notation  NI(0,cr  ) 

B 

means  normally  and  independently  distributed  with  mean  0  and 
2 

variance  cr  . 

The  assumption  of  a  normal  distribution  for  observations 
on  oil  sample  burns  seems  reasonable  and  is  documented  by 
previous  work  with  oil  analysis  data  (references  4,  5  and 

6) .  The  assumption  of  independence  is  somewhat  questionable 
based  on  previous  work.  It  has  been  found  that  there  exists 
a  dependence  between  certain  elements  and  a  dependence 
between  PPB  concentrations  (references  4,  5  and  6). 
However,  in  view  of  the  fact  that  all  of  the  analyses  in 
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this  report  were  performed  for  one  ppa  concentration  and  one 
element  at  a  time,  the  assumption  of  independence  does  not 
seee  unreasonable. 

The  suss  of  squares  foe  the  nodal  (reference  7)  are: 

2 

ssa  -  ijk (7  )  , 


2  2 

ssa  «  jk+sum  (Y,  -  7  )  =  sua,  {I,  )  /jk  -  ssa 

i  i . .  ...  i  l •  • 


ss(Q|A)  -  K-esua.  (T.  -  r,  ) 

r ,  j  i.  j  •  i». 


sua.  (y  )  /k  -  sua  (y  )  /jk  , 

i,j  13.  i  i.  . 


SSE  *  sua  (Y  -  7  ) 

i,j»k  ijk  ij. 


2  2 
sua,  J  y,  -  sua,  (y  )  /k, 
i,j,k  ijk  i, j  ij. 


SSTO  *  sua  Y 

ir  j,  1c  ijk 


i  *  i  j  55  1 ,  • . . ,  J  /  k  =*  1,  . . .  ,  K  ( a  1 ) 

where  SSM  is  the  sun  of  squares  due  to  the  aean  effect, 

SSA  is  the  sua  of  squares  due  to  the  laboratory  effect, 

SS (5 1  A)  is  the  sun  of  squares  due  to  the  day  within 
spectrometer  effect, 


SSE  is  the  sun  of  squares  due  to  the  random  error 
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9 


affect, 

SSTO  is  the  total  sub  of  squares. 


SOM  Y.  /IJK, 
irj #k  ijk 


SOM.  Y  /  JK 
j,k  ijk 


7  «  SOM  Y  /K  ,  ir  i  and  k  are  defined  in  equation  (al) , 

ij.  k  ijk 


I  is  the  number  of  laboratories,  J  is  the  nunber  of  days 
sampled,  and  K  is  the  number  of  replications  for  each 
laboratory  and  day  combination. 


The  mean  squares  for  the  model  (reference  7)  are: 


MSA  =  SSA/  (I-  1)  , 

MS  (B  |  A)  =•  SS  (B  |  A)  /I  (J-  1)  ,  and 


MSE  *  SSE/IJ  (1-1 )  (a2) 

vhere  MSA  is  the  mean  squares  due  to  the  spectrometer 
effect. 


MS  (B |  A)  is  the  maan  squares  due  to  the  day 
(standardization)  within  spectrometer  effect,  and 

MSE  is  the  mean  squares  due  to  random  error. 

The  two  test  statistics  accociated  with  the  model  are: 

F  *  MSA/MS  (B|  A)  with  (1-1)  and  I(J-1)  degrees  of 
A 

freedom  and 


F  =  MS  (B1  A) /MSE  with  I  (J-1)  and  IJ  (K-1)  degrees  of 
B 
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freedom,  (a3) 

where  F  is  tha  test  statistic  used  to  test  for  a 

A 

significant  spectroaeter  effect  and 


F  is  the  test  statistic  used  to  test  for  a  significant 
8 

day  effect. 


The  usual  estimators  for  the  conpouents  of  variance  are: 

A 

a 

<1  *  MSB, 

A 

2 

a  =»  (US  (B |  A)  -  aSE)/K,  and 
B 

A 

2 

a  *  (asA  -  as(B|A))/JK.  (ah) 

A 
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APPENDIX  B 


ESTIMATORS  FOR  AI  AND  HI 


While  developing  bounds  on  AI  foe  the  three  cases,  it 
was  found  that  for  variance  estiaates  the  usual  chi-square 
statistic  was  not  appropriate.  The  sum  of  k  independent 

2  2 

chi-square  distributed  random  variates  (n  3  /a  )  with  n 

i  i  i 


degrees  of  freedoa 
2  2 

variate  (nS  /a  ) 


(reference  81  is  a  chi-square  random 

with  n  =  a  +■  n  * . +  n  degrees  of 

i2  k 


freedom.  However,  the  variance  estiaates  used  in 

determining  the  bounds  for  AI  were  linear  combinations  of 
variances  for  which  Cochran's  theorem  is  not  applicable. 
Using  the  procedures  described  in  chapter  17  of  reference  8, 
an  approximate  chi-square  statistic  can  be  formed  as 
follows: 


u  3  ng/G 


2  2  2 

where  n  3  (SUM  g  x  )  /SUM  (g  x  /a  )  ,  i 

i  i  i  i  i  i  i 


approximate  degrees  of  freedoa. 


(bl) 

1 ,.  . .  ,  K,  is  the 


a  =  v*  *  *  V  ' 
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g  -  g  ESA  ♦  g  &S(E|A|  ♦  g  HSE, 

12  3 

x^  =  tiSA,  »  as  (B|  A)  V  *3  *  8SE  and.  the  g.'s  will  be 

defined  in  the  development  that  follows  for  the  variance 
estimates. 

A.  CASE  I 


A  95  percent  confidence  bound  on  All  (|u  -  y  I)  ,  is 

ij- 

1/2 

given  by  t  ( . 975) g  , 

n 


where  y  is  the  sample  lean  of  K  observations  with  the  ith 

ij* 

spectrometer  on  day  j. 


g  is  a  variance  estimate  of  (y  -  7,  ) 


n  is  the  approximate  degrees  of  freedom,  and 


t  (.975)  is  the  tabled  value  of  Student's  t  distribution 


having  n  degrees  of  freedom. 


The  variance  of  (y  -  T  )  can  be  found  as  foLlows; 

ij- 


V(y-T  )  =  V(Y  )  a  7(SDB  I  /K) 
ij.  ij.  k  ijk 


2 

=  (1/K  )  7  (SOM  (U  ♦  A.  «•  B  *  E  )  ) 

k  i  i>  ijk 
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■» 


2  2  2  2  2 

=  1/K  (K  a  *  K  a  +  SOM  V(E  ) 
A  B  k  ijk 


2  2  2 

=  a  +  a  *■  a  /15,  k  =  1,...,K 
A  B 


An  estimate,  g,  of  V{  u  -  7  )  is  (from  equation  (a4)): 

ij. 


g=  g^MSA  +  g2»S(B|A)  ♦  g^BSE, 


g  =  1/JK,  g^  =  (J-1)/JK  and  g3  =  (K-H)  /Kti,  (b2) 

where  I  is  the  number  of  spectrometers  used  in  the  analysis. 


J  is  the  number  of  days  (2)  , 


K  is  the  number  of  .epeated  observations,  and 


H  is  the  number  of  sample  burns  for  which  tables  wero 
constructed  (10  or  15)  . 


The  approximate  degrees  o£  freedom  n  associated  with  the 
variance  estimate  g  is: 


n  *  [g  USA  +  g  ifS(BU)  +  g  USB]/ 

X  2  3 

z  z  z  z  z  z 

[g ,  (HSA)  /(I-1)  ♦  g  (MS  (B  |  A)  )  /I(J-1)  +  g  (BSE)  /IJ(K-I)] 

12  3 

(b3) 

The  usual  95  percent  upper  confidence  bound  on  ail  would 
2  1/2 

b9  [  a  (B  SE)  /  x  (-05)  ]  •  where  n  is  the  degrees  of  freedom 

n 

2 

associated  with  the  variance  estimate,  as.E,  and  x  (-05)  is 

n 
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the  tabled  value  of  a  chi- square  distribution  having  n 

degrees  of  freedom.  During  analysis  it  was  found  that  this 
upper  bound  on  ail  was  not  realistic  when  compared  with  the 
saaple  standard  deviations  computed  from  the  data.  k 
Bartlert  test  for  equal  variances  was  performed  (reference 
9)  for  several  PPM  and  element  coabinations.  It  was  found 
that  for  all  coabinations  tested,  the  hypothesis  of  equal 
variances  was  rejected.  Therefore,  another  aethod  for 
estimating  the  upper  bound  on  HI1  was  developed. 

To  estimate  the  upper  bound  on  RI1,  the  sample  standard 
deviations  for  both  days  on  all  spectrometers  were  used  to 
construct  an  empirical  cumulative  distribution  function. 
The  95  percent  upper  bound  was  found  by  linear  interpolation 
on  the  empirical  cumulative  distribution  function. 

To  find  the  upper  bound  on  Ell  for  H  =  (0  saaple  burns, 
it  is  assumed  that  the  upper  tail  of  the  distribution  of 
saaple  variances  can  be  approximated  by  the  upper  tail  of  a 
normal  distribution.  To  develop  a  relation  between  the 
distributions  foe  10  and  15  sample  burns,  it  is  further 
assumed  that  the  saaple  variances  have  first  and  second 
moments  that  are  related  in  the  same  manner  as  those  of 
chi-square  variates.  The  variance  of  a  chi-square  variate 

2  2 

(v  s  /  a  )  is  2v  (reference  7),  and 
i  i  i 

2  4  2  2  2  4 

V(SJ  =  cr  /v^V  (V^/  CT  )  =  2  a 
and,  from  the  95  percent  upper  bound  we  have 

2  2 

P  (S  £  RI  )  -  .95 

i  l 

22  4  1/2  22  4  1/2 

or,  P((St-  a  )  /( 2  a  /vj  <  (H^-  a  )/(2  a  /vj  )  =  .95 
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2  2  ♦  1/2 
or,  ®((aii-a  )  / (2  a  /vj  )  =  . 

2 

where  is  the  estimate  of  sample  variance  for  H  «  15 

sample  barns, 

v  is  the  degrees  of  freedom  of  the  estimate  (v  *  14), 
i  i 

and 

0  is  the  standard  normal  CDF . 

After  equating  the  above  to  a  similar  statement  for  N  =  10 

sample  burns,  we  have: 

t/2  2  1/2  2  1/2 

aii  =  ( (14/9)  an  -  ( i- { 14/9)  )  a  ) 

2 

Heplacing  a  by  its  estimate  (MSE)  gives  the  upper  bound  on 
ail  for  N  3  10  sample  burns. 

B.  CASE  IX 


A  95  percent  confidence  bound  on  AI2  (|y  -  x|)  is  given 
1/2 

by  t  (.975)  g  ,  where  y  is  the  sample  mean  of  the  first 
n 

set  of  H  observations,  and  z  is  the  sample  mean  of  the 
second  set  of  N  observations. 


The  variance  of  (T  -  Y  )  can  be  found  as  follows: 

ij.  i j *  . 
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7(Y  -  Y  )  =  V(Y.  ,  )  +  7(7  )  -  2Cov  (Y  ,Y  ) 

ij.  ij'.  13«  ij  •  lJ*  * 


C0*(T.  .  ,T  )  =  E(I,  -»)  -  «  - 

13.  ij'.  IJ*  i3  •  4 

Hence,  7  (Y.  -  7. 

ij-  i}'  * 


2  2 
)  =  2  c  *  2  cf  /15. 
B 


An  estimate,  g,  of  7(7  -  Y  )  is  (from  agnation  (a4) ) 

i  J  •  i3  * 


g  =  g  MS  A  +  g  MS  (B  1  A)  ♦  g  MSB 

7  ^4  ^  9  J 


g  •  0,  g  *  2/K ,  and  g  =  2  (K-H) /XN  (b4) 

12  3 

The  approximate  degrees  of  freedom,  n,  ara  found  using 
equation  (b 3)  with  the  g.'s  as  defined  in  equation  (b4). 


z  2 

A  95  percent  confidence  interval  on  812  (s  /s  )  is  given 

y  ^ 

byj. 


1/P  (.975)  <  HI2  <  F  (.975)  <b5> 

n*,u2  a‘,n* 

where  P  (.975)  is  the  tabled  value  of  an  P  distribution 

ul,n2 

having  nl  and  n2  degrees  of  freedom. 


Since  the  number  of  observations,  H ,  is  the  same  for 
both  sets  of  observations  nl  =  n2  =  H-1  -  9  or  14. 
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C.  CASE  III 


A  95  percent  bound  on  M3  <{y  -  x|)  is  given  by 

l/a 

t  <.9?S )  q  , 
n 

where  f  is  the  sacpi*  taeau  of  tho  control  laboratory’s  M 
observations ,  and 


1  is  the  sample  mean  of  the  given  laboratory's  N 
ob  serva  tioau . 

The  variance  or  (Y ,  -  Y,  )  is  found  as  follows: 

ij.  1'  j'  . 


V(Y  -  Y  ) 

ij-  i'J'. 


*  V(Y  )  ♦  V  (Y  I  -  2C0V  (Y  ,Y  ) 

ij.  1* j  ' .  ij .  i'  j  »  . 


Cov(Y  ,  ,Y  )  «  0 
i5*  i'J'* 


V(Y  -I  )  -  2  o  ♦  2  a  ♦  2  a  /15 
i  j*  i'  j '  .  A  B 


An  estimate, 


g,  of  V  (*  -  Y  )  is  (from  equation  <a4) ) ; 

i  j.  i '  j  '  • 


g  ■  g  &SA  +  g  MS  (b  |A)  ♦  g  MSE, 

&  2  9 


42 


2(J-1)  /JK, 


2  (K-H)  /KN 


gt  *  2/ J K t  g2  - 


and  g  = 

3 


(b6) 


The  appropriate  degrees  of  freedom,,  n,  are  found  using 
equation  (b3)  with  the  g,*s  as  defined  in  equation  (b6)  . 

The  CJ5  percent  bound  on  SI3  is  found  using  equation  ( b 5)  « 
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iPPENDIX  C 


COEEELATION 


During  analysis  of  the  data  provided  by  the 
participating  laboratories,  the  sample  standard  deviations 
were  computed  and  the  correlation  coefficient  (reference  3) 
vas  computed  as  follows; 

rnum  *  SOM  x  y  -  SUM  x  SUE  y/N 
i  i  i  i  i  i 

2  2  2  2  1/2 
rdaaom  *  ( (SUE  x  -(SUB  x  )  /N)  (SUM  y  -(SUE  y  )  /N)  ) 
ii  li  r  i  ii 

r  *  rnum/rdenom  ,  i  *  1,...,I  (cl) 

where  y  is  the  sample  standard  deviation  computed  from  the 
i 

data  collected  on  the  first  day  that  data  was  collected  for 
each  PPE  and  element  combination, 

x  is  the  sample  standard  deviation  computed  from  the 
i 

data  collected  on  the  second  day,  and 


I  is  the  number  of  Laboratories  submitting  data  for  a 
given  PPE  concentration. 

Table  VII  gives  the  values  of  the  correlation 
coefficient  r  for  each  PPE  and  element  combination.  To 
determine  whether  the  correlation  coefficents  compute^  were 
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significant,  the  test  statistic  t  vas  computed  as  follows: 

1/2  2  1/? 

t  -  r  (I  -  2)  /{I  -  c  )  (5) 

Under  the  hypothesis  that  the  correlation  is  zero,  t  is 
distributed  as  Student's  t  distribution  with  1-2  degrees 
of  freedom  (reference  9).  Entering  a  t  distribution  table 
with  the  test  statistic  t  and  1-2  degrees  of  freedom 
yielded  the  tail  areas  that  are  also  given  in  Table  VII.  If 
one  chose  .05  as  the  desired  level  of  significance,  any  tail 
area  listad  in  Table  VII  not  greater  than  .05  would  imply 
that  the  corresponding  correlation  coefficient  is 
significant:  there  is  a  significant  positive  correlation 
between  the  standard  deviations  of  the  first  and  second 
day's  observations  at  a  given  laboratory.  The  discussion 
above  assumes  that  the  day  one  and  day  two  sample  standard 
deviations  are  observations  on  a  bivariate  normal  population 
(reference  9) .  However,  sample  standard  deviations  are  not 
normally  distributed.  Hence,  the  results  in  table  VII 
should  only  be  interpreted  as  rough  indicator:,. 
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APPENDIX  D 


PARTICIPATING  LABORATORIES 


the  following  is  a  list  of  the  laboratories  that 
cooperated  by  making  the  the  sample  barns  and  submitting 
their  results  for  the  author's  analyses.  Tha  numbering 
system  has  no  relation  to  the  oil  analyses  program. 

Lab  no.  Location  of  laboratory 

1  Technical  Support  Center,  Pensacola 

2  McClellan  APB 

3  Hunter  AAF 

4  NAS  Norfolk 

5  M CAS  Cherry  Point 

6  Langley  AFB 

7  Pease  AFB 

S  Laughlin  AFB 

9  Bergstrom  AFB 

10  NAS  Whidbey  Island 

11  Dover  AFB 

12  FT  Rucker 

13  Dyess  AFB 

14  Myrtle  Beach  AFB 

15  ANG,  Jacksonville,  FI 

16  Davis  Monthan  AFB 

17  McChord  AFB 

18  Shaw  AFB 

19  Eglin  AFB 

20  Holloman  AFB 

21  Randolph  AFB 

22  Hinot  AFB 
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23 

NAS  Alaaeda 

24 

Ft  Hood 

25 

Grandf ork 

AFB 

26 

£11  sworth 

AFB 
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